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The U.S. Office of Management and Budget (OMB) developed 
the Program Assessment Rating Tool (PART) in 2002 to assess 
performance of federal programs, develop program 
improvement plans, and link program performance to budget 
decisions. PART strengthened the Government Performance and 
Results Act (GPRA) of 1993 by including in the PART 
evaluation process the program goals and performance measures 
developed under GPRA by the different Federal programs. 
Under this system, program managers and their advocates must 
develop performance goals that focus on program outcomes and 
develop valid instruments to measure outcomes accurately. This 
latter area is our concern. 

At the end of 2005, OMB had reviewed approximately 80% of 
federal programs and had projected completion of the remaining 
program assessments by the end of 2006 or early 2007 
(ExpectMore.gov, n.d.). The Older Blind Independent Living 


Program, initially funded in FY1987 by the Rehabilitation 
Services Administration (RSA), will be among the last of the 
programs reviewed by OMB. This program provides services to 
individuals aged 55 and older whose significant visual 
impairment makes competitive employment very difficult, but 
for whom independent living goals are feasible. The program 
serves approximately 62,000 consumers nationally through 56 
independent living agencies in states and territories (Moore, 
Steinman, Giesen, & Frank, 2006). The state independent living 
programs offer blindness and low-vision services such as 
training in orientation and mobility, communications, and daily 
living s kill s. 

RSA requires that independent living programs annually report 
demographic information on consumers receiving services and 
the numbers receiving specific types of services. Although some 
states collect information on consumer outcomes (for example, 
improvement in daily living skills), RSA does not request such 
data. After being vacant for the last three years, the RSA 
independent living program manager position is currently being 
filled as of July 2006. We expect that a major priority of the 
new manager will be the development of program goals and 
performance measures that can be used to demonstrate program 
effectiveness. 

RSA has supported two national surveys of functional outcomes 
and consumer satisfaction among those served by the Older 
Blind Independent Living Program (for example, Moore et al., 
2006). The most recent of these surveys was conducted in 2004. 
Although its data may be too old to be used in any OMB 
evaluation, perceived functional outcomes were highly positive 
and overall satisfaction of consumers was high (over 90%). 
Results from the evaluation instrument were reported mostly on 
an individual-item basis. Although the survey questionnaire was 
carefully developed, its dimensionality and psychometric 
properties as an instrument have not been intensively 
investigated. It is important that any assessment tool used to 



collect outcome data be continually evaluated and refined. In the 
current political climate, where disability-specific programs 
continue to be at risk (F. Schroeder, personal communication, 
April 26, 2006), survival of programs serving blind and visually 
impaired consumers may depend on the initiative and rigor of 
administrators in demonstrating that their programs are 
effective. Thus, the present study focuses on improvement of 
measurement quality and consideration of additional 
methodological approaches to evaluation of national services to 
persons who are blind or visually impaired. 

In addition to traditional methods of instrument development 
(for example, internal consistency reliability analysis, 
exploratory factor analysis), covariance structure analysis (CSA) 
or structural equation modeling (SEM) was employed. SEM 
methods have a variety of advantages, including rigorous 
evaluation of a measurement factor model, ability to address 
item measurement error correlations, and ability to address 
direct and indirect effects in a structural model (Kelloway, 

1998). A secondary purpose of this study is to demonstrate how 
CSA can contribute substantively to the examination of 
relationships among measured constructs, specifically program 
effectiveness: how improvements in blindness and low-vision 
rehabilitation skills can positively affect other activities of daily 
living (ADL) related to functioning and satisfaction, as 
suggested by Cavenaugh and Steinman (2005) and implied by 
general rehabilitation theory (Arokiasamy, 1993). 

Method 

Data source and instrument 

Data were from a national survey of functional outcomes and 
satisfaction among 1,025 consumers in the Older Blind 
Independent Living Program (Moore et al., 2006). Due to 
respondents' considerable use of the "Not Applicable" 
alternative, complete data were available for 603 respondents on 
the 10 survey items of interest. The survey instrument was 



designed to obtain information on perceived outcome and 
satisfaction specific to the impact of independent living services 
on older individuals who are blind and are served under Title 
VH-Chapter 2. Items were developed with the assistance of an 
expert panel, with attention to domain content validity, and were 
pretested in four states. 

The specific wording for the items and response scales is given 
in Table 1 . Based on face validity, items were intended to tap 
domains of respondents' confidence in areas of activity and 
mobility: ADLs (Items 1-4); print access or vision use (Items 5- 
6); community integration (Item 7); and satisfaction (Items 8- 
10). The entire scale was subjected to exploratory factor 
analysis, internal consistency reliability analysis of factors, 
confirmatory factor analysis, and latent factor modeling using 
LISREL 8.7, a statistical program (Joreskog & Sorbom, 2005). 

Results 

Distributional examination of variables indicated acceptable 
results except for some negative skew for Items 8 and 9 
(Satqua8 and Satim9). Multivariate normality on the set of 
variables was tested using PRELIS 2.7, a companion program to 
LISREL 8.7 (Joreskog & Sorbom, 2005). The resulting Mardia's 
coefficient of 1.49 indicated that multivariate normality was not 
violated. Exploratory principal components analysis and 
principal axis factoring using oblimin (nonorthogonal) rotation 
yielded three factors: general ADL (Items 1-4, 7) satisfaction 
(Items 8-10), and vision use (Items 5-6). Cronbach's alphas 
were .80, .79, and .77, respectively, with very little change in 
alpha if specific items in the scale set were deleted. 

CSA typically involves development of a "measurement model" 
and then specification of a structural model that describes 
linkages between the (latent) factors. Using correlation matrix 
input, a three-factor measurement model (or confirmatory factor 
analysis) was specified using Items 1-4 and item 7 as reflective 


indicators of ADL improvement (ADLIMP), Items 5-6 for 
vision use improvement (READ), and Items 8-10 for 
satisfaction with independent living program services 
(SATISFAC). An advantage of the CSA model is that the 
measurement error of an indicator can be separated from its 
variance explained by the factor on which it has a substantial 
factor loading. In addition, correlations between item 
measurement errors also can be assessed. 

Item 2 (Move2) was dropped from the measurement model 
analysis because of cross-loading tendency and error correlation 
with Item 1 (Prfactl). Further examination of diagnostics 
(standardized residuals and modification indices) indicated that 
the fit of the confirmatory factor model could be improved. 
Measurement errors for Items 8 and 9 (Satqua8 and Satim9), 
and for Items 3 and 4 (Meal3 and Houkep4) were allowed to 
correlate because of item wording and logical similarities. The 
final measurement model was a first-order confirmatory factor 
analysis model with correlated factors: SATISFAC - ADFIMP 
= .68, SATISFAC - READ = .36, ADFIMP - READ = .59, all p 
< .05. Construct reliabilities for SATISFAC, READ, and 
ADFIMP were .77, .77, and .75, respectively. 

Several structural models were examined, including a primary 
and two alternative models. (Another advantage of CSA is that 
the regression relationships between factors in the structural 
model are analogous to relationships between "true scores" on 
the factors; the relationships have been corrected for attenuation 
due to measurement error.) The primary interest Model 1 
postulates that vision-use improvement (READ, £1) directly 
influences the endogenous latent constructs of ADF 
improvement (ADFIMP, rj 2) and satisfaction (SATISFAC, rj 1), 
and that ADF improvement in turn influences satisfaction. The 
final standardized Model 1, with measurement errors, loading, 
and regression path estimates shown in Figure 1, achieved good 

fit: x 2 (22) = 60.94, p < .0001; RMSEA = 0.0542; p for Close Fit 
(RMSEA < .05) = .311; GFI = 0.978; AGFI = 0.955. The only 


nonsignificant path was to SATISFAC from READ - - 
0.07). 

Overall, 47.2% of variance in program satisfaction and 34.8% of 
the variance in ADL improvement were explained by the model. 
All of the effect of vision-use improvement on program 
satisfaction was indirect, mediated through ADL improvement. 

An alternative Model 2 was the same as Model 1 except that 
there was no path between ADL improvement and program 
satisfaction. This model was examined to consider results that 
might be obtained from a simple multivariate regression model 
(READ predicting SATISFAC and ADLIMP) without an 

indirect effect path. Model 2 fit was unacceptable: x 2 (23) = 
164.3, p < .00001; RMSEA = 0.101; p for Close Fit (RMSEA 

< .05) = .000; GFI = 0.943; AGFI = 0.888. Diagnostics (largest 
modification indices) indicated the need for a path between 
ADL improvement and program satisfaction in either direction. 

Alternative Model 3 was the same as Model 1 except that the 
direction of the indirect effect path was intentionally reversed: 
to ADL improvement from program satisfaction— a possibility 
suggested by the modification indices obtained from Model 2. 
Model 3 fit was good, with fit statistics identical to those for 
Model 1, as were all measurement model parameter estimates. 
However, the structural path coefficients were quite different: 
y n (SATISFAC from READ) = .36, y 21 (ADLIMP from 
READ) = .39, and (ADLIMP from SATISFAC) = .54, all p 

< .05. 

Discussion 

Based on exploratory factor analysis, the independent living 
survey instrument was found to measure three factors: 
satisfaction with program services, general improvement in 
ADLs, and improvement in vision use or reading. Factor alpha 
reliabilities were acceptable. Little other useful information 
regarding instrument refinement or improvement was gleaned 



from traditional reliability analysis. 

Covariance structure analysis did provide additional useful 
information regarding the instrument and substantive 
interpretation of the relationships between constructs measured 
by the instrument. The loadings of the items on the factors and 
their error variances can help distinguish those items that are 
doing a good job measuring the factor or construct they reflect 
from those that have high measurement error or are measuring 
some unknown factor. For example, as shown in Figure 1, most 

of the variance in item 5 (managing paperwork— loading of .84 2 
= 71% — is associated with the vision-use improvement factor, 
indicating a satisfactory item. In contrast, item 4 (managing 
housekeeping tasks)-loading of .632 = 40%-has less than half 
of its variance associated with the ADL improvement factor, 
suggesting that the item is weak because 60% of its variance is 
associated with error of measurement or other unknown 
influences. Such information is useful in targeting items for 
improvement. Based on high item-error variance and tendency 
for an item to load on more than one factor (cross-loading), we 
recommend that the item dealing with being "better able to 
move confidently around where I live" be removed from the 
instrument, possibly to be replaced with an item that is likely to 
be more effective in measuring ADL improvement. Other items 
with loadings of less than .7 also are candidates for 
improvement or replacement. 

Measurement model analysis also provides information about 
redundant items as indicated by correlations between their errors 
of measurement. We found this tendency for satisfaction in 
Items 8 (service quality) and 9 (timeliness), and independent 
living Items 3 (prepare meals) and 4 (manage housekeeping 
tasks), which should be revised to avoid similar (redundant) 
responding tendencies. 

Covariance structure analysis has limitations. Coovert, Penner, 
and MacCallum (1990) characterized the goodness-of-fit 



information provided by this methodology as indicating the 
(statistical) plausibility of a model. In addition to confirmation 
of Model 1 (Figure 1), our analysis indicates that our data also 
fit (confirm the plausibility) of a causally irrational model that 
postulates improvement in vision use directly affects 
satisfaction, which, in turn, affects ADL improvement. Clearly, 
covariance structure analysis is a useful analytic tool, but 
plausible theory is an essential accompanying component for 
interpretation of results. 

Summary 

In summary, the structural relationships between the latent 
constructs measured by the instrument provide useful 
substantive information beyond measurement issues. Our 
postulated conceptual model was that improvement in print 
access and vision-related tasks— presumably from independent 
living services— affected ADL functioning and satisfaction with 
services. This model is derived from a general theory of 
rehabilitation in which interventions are expected to have 
positive functional effects for the individual (Arokiasamy, 

1993). Model results support the theory, indicate that the 
national Older Blind Independent Living Program is effective, 
and suggest that the vision-related intervention effect is not 
direct, but mediated. Vision-use improvement affects 
improvement in ADL functioning, which, in turn, leads to 
consumer satisfaction. If we had found that there was no 
significant effect (path) of improvement in vision use on ADL 
functioning, or no effect from ADL functioning to satisfaction, 
program effectiveness might be questioned. (We have seen such 
data [Giesen, 2006].) Thus, by thinking of (and modeling) the 
process by which independent program services affect 
satisfaction, we hope to add an additional dimension to 
measurement of program effectiveness and contribute to efforts 
that add a greater understanding of how and why program 
interventions work. 



References 


Arokiasamy, C. V. (1993). A theory for rehabilitation? 
Rehabilitation Education, 7, 77-98. 

Cavenaugh, B. S., & Steinman, B. A. (2005). Estimated number 
of persons eligible for vision rehabilitation services under 
expanded Medicare. Journal of Visual Impairment & 
Blindness, 98, 431-436. 

Coovert, M. D., Penner, L. A., & MacCallum, R. (1990). 
Covariance structure modeling in personality and social 
psychological research. In C. Hendrick & M. S. Clark (Eds.), 
Review of personality and social psychology, vol. 11: Research 
methods in personality and social psychology (pp. 185-216). 
Newbury Park, CA: Sage. 

ExpectMore.gov (n.d.). Expect federal programs to perform 
well, and better every year. Retrieved May 2, 2006, from 
http://www.whitehouse.gov/omb/expectmore/about.html 

Giesen, J. M. (2006). Preliminary analyses for state-level older 
blind independent living program evaluations. Unpublished 
raw data. 

Joreskog, K. G., & Sorbom, D. (2005). LISREL 8.72. Chicago: 
Scientific Software International. 

Kelloway, E. K. (1998). Using LISREL for structural equation 
modeling: A researcher's guide. Thousand Oaks, CA: Sage. 

Moore, J. E., Steinman, B. A., Giesen, J. M., & Frank, J. J. 
(2006). Functional outcomes and consumer satisfaction in the 
Older Blind Independent Living Program. Journal of Visual 
Impairment & Blindness, 100(5), 285-294. 

J. Martin Giesen , Ph.D., professor, Department of Psychology, 

and senior research scientist, Rehabilitation Research and 


Training Center on Blindness and Low Vision (RRTC-BLV), 
P.O. Drawer 6189, Mississippi State University, Mississippi 
State, MS 39762; e-mail: < imgl(a)ra.msstate.edu >. Brenda S. 
Cavenaugh , Ph.D., CRC, director of research and associate 
research professor, RRTC-BLV, Mississippi State University; e- 
mail: < bcavenaugh(d)colled.msstate.edu >. 


IS 


Download braille-ready file 

Download ASCII text file (ASCII files are for download 


only) 

•T 

‘A** Download PDF file 

Previous Article | Next Article | Table of Contents 

JVIB, Copyright © 2006 American Foundation for the Blind. All 

rights reserved. 


Search JVIB | JVIB Policies | Contact JVIB | Subscriptions | 

JVIB Home 

If you would like to give us feedback, please contact us at 

jvib@afb.net . 


www.afb.org | Change Colors and More [ Contact Us 

Go 


Site Search 


Site Map 


About AFB | Press Room | Bookstore | Donate | Policy Statement 


Please direct your comments and suggestions to afbinfo@afb.net 
Copyright © 2006 American Foundation for the Blind. All rights reserved. 


